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Abstract 

Many machine learning tasks can be ex- 
pressed as the transformation — or transduc- 
tion — of input sequences into output se- 
quences: speech recognition, machine trans- 
lation, protein secondary structure predic- 
tion and text-to-speech to name but a few. 
One of the key challenges in sequence trans- 
duction is learning to represent both the in- 
put and output sequences in a way that is 
invariant to sequential distortions such as 
shrinking, stretching and translating. Recur- 
rent neural networks (RNNs) are a power- 
ful sequence learning architecture that has 
proven capable of learning such representa- 
tions. However RNNs traditionally require 
a pre-defined alignment between the input 
and output sequences to perform transduc- 
tion. This is a severe limitation since finding 
the alignment is the most difficult aspect of 
many sequence transduction problems. In- 
deed, even determining the length of the out- 
put sequence is often challenging. This pa- 
per introduces an end-to-end, probabilistic 
sequence transduction system, based entirely 
on RNNs, that is in principle able to trans- 
form any input sequence into any finite, dis- 
crete output sequence. Experimental results 
for phoneme recognition are provided on the 
TIMIT speech corpus. 

1. Introduction 

The ability to transform and manipulate sequences is 
a crucial part of human intelligence: everything we 
know about the world reaches us in the form of sen- 
sory sequences, and everything we do to interact with 
the world requires sequences of actions and thoughts. 
The creation of automatic sequence transducers there- 
fore seems an important step towards artificial intel- 
ligence. A major problem faced by such systems is 
how to represent sequential information in a way that 
is invariant, or at least robust, to sequential distor- 



tions. Moreover this robustness should apply to both 
the input and output sequences. 

For example, transforming audio signals into sequences 
of words requires the ability to identify speech sounds 
(such as phonemes or syllables) despite the apparent 
distortions created by different voices, variable speak- 
ing rates, background noise etc. If a language model 
is used to inject prior knowledge about the output se- 
quences, it must also be robust to missing words, mis- 
pronunciations, non- lexical utterances etc. 

Recurrent neural networks (RNNs) are a promising ar- 
chitecture for general-purpose sequence transduction. 
The combination of a high-dimensional multivariate 
internal state and nonlinear state-to-state dynamics 
offers more expressive power than conventional sequen- 
tial algorithms such as hidden Markov models. In 
particular RNNs are better at storing and accessing 
information over long periods of time. While the early 
years of RNNs were dogged by difficulties in learn- 
ing (Hochreiter et al., 2001), recent results have shown 
that they are now capable of delivering state-of-the-art 
results in real-world tasks such as handwriting recog- 
nition (Graves et al., 2008; Graves & Schmidhuber, 
2008), text generation (Sutskever et al., 2011) and lan- 
guage modelling (Mikolov et al., 2010). Furthermore, 
these results demonstrate the use of long-range mem- 
ory to perform such actions as closing parentheses after 
many intervening characters (Sutskever et al., 2011), 
or using delayed strokes to identify handwritten char- 
acters from pen trajectories (Graves et al., 2008). 

However RNNs are usually restricted to problems 
where the alignment between the input and output 
sequence is known in advance. For example, RNNs 
may be used to classify every frame in a speech signal, 
or every amino acid in a protein chain. If the network 
outputs are probabilistic this leads to a distribution 
over output sequences of the same length as the input 
sequence. But for a general-purpose sequence trans- 
ducer, where the output length is unknown in advance, 
we would prefer a distribution over sequences of all 
lengths. Furthermore, since we do not how the inputs 
and outputs should be aligned, this distribution would 
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ideally cover all possible alignments. 

Connectionist Temporal Classification (CTC) is an 
RNN output layer that defines a distribution over all 
alignments with all output sequences not longer than 
the input sequence (Graves et al., 2006). However, as 
well as precluding tasks, such as text-to-speech, where 
the output sequence is longer than the input sequence, 
CTC does not model the interdependencies between 
the outputs. The transducer described in this paper 
extends CTC by defining a distribution over output 
sequences of all lengths, and by jointly modelling both 
input-output and output-output dependencies. 

As a discriminative sequential model the transducer 
has similarities with 'chain- graph' conditional random 
fields (CRFs) (Lafferty et al., 2001). However the 
transducer's construction from RNNs, with their abil- 
ity to extract features from raw data and their poten- 
tially unbounded range of dependency, is in marked 
contrast with the pairwise output potentials and hand- 
crafted input features typically used for CRFs. Closer 
in spirit is the Graph Transformer Network (Bottou 
et al., 1997) paradigm, in which different iable mod- 
ules (often neural networks) can be globally trained 
to perform consecutive graph transformations such as 
detection, segmentation and recognition. 

Section 2 defines the RNN transducer, showing how 
it can be trained and applied to test data, Section 3 
presents experimental results on the TIMIT speech 
corpus and concluding remarks and directions for fu- 
ture work are given in Section 4. 

2. Recurrent Neural Network 
Transducer 

Let x = (x±, #2, • • • , xt) be a length T input se- 
quence of arbitrary length belonging to the set X* 
of all sequences over some input space X. Let y = 
(i/i , 2/2 , . . . , yu) be a length U output sequence belong- 
ing to the set y* of all sequences over some output 
space y. Both the inputs vectors x t and the output 
vectors y u are represented by fixed-length real- valued 
vectors; for example if the task is phonetic speech 
recognition, each x t would typically be a vector of 
MFC coefficients and each y t would be a one-hot vec- 
tor encoding a particular phoneme. In this paper we 
will assume that the output space is discrete; however 
the method can be readily extended to continuous out- 
put spaces, provided a tractable, differentiable model 
can be found for y. 

Define the extended output space y as yi)0, where 
denotes the null output. The intuitive meaning of 
is 'output nothing'; the sequence (2/1, 0, 0, 2/2, 0, 2/3) G 



y* is therefore equivalent to (2/1, 2/2? 2/3) £ 3^*- We 
refer to the elements a G ^P* as alignments, since the 
location of the null symbols determines an alignment 
between the input and output sequences. Given cc, 
the RNN transducer defines a conditional distribution 
Pr(a G y*\x). This distribution is then collapsed onto 
the following distribution over y* 

PT(yey*\x)= Pr H*0 (i) 

where B : y* \-> y* is a function that removes the null 
symbols from the alignments in y* . 

Two recurrent neural networks are used to determine 
Pr(a G y*\x). One network, referred to as the 
transcription network T , scans the input sequence x 
and outputs the sequence / = (/i,...,/t) of tran- 
scription vectors 1 . The other network, referred to 
as the prediction network Q, scans the output se- 
quence y and outputs the prediction vector sequence 
9 = (00,01..., 9u)- 

2.1. Prediction Network 

The prediction network Q is a recurrent neural network 
consisting of an input layer, an output layer and a 
single hidden layer. The length U + 1 input sequence 
y = (0, 2/1, • • • , yu) to Q output sequence y with 
prepended. The inputs are encoded as one-hot vectors; 
that is, if y consists of K labels and y u = k, then 
y u is a length K vector whose elements are all zero 
except the k th , which is one. is encoded as a length 
K vector of zeros. The input layer is therefore size 
K. The output layer is size K + 1 (one unit for each 
element of y) and hence the prediction vectors g u are 
also size K + 1. 

Given y, Q computes the hidden vector sequence 
(ho, . . . , hu) and the prediction sequence (go, . . . , gu) 
by iterating the following equations from u = to U: 

h u = H (W ih y u + W hh h u -! + b h ) (2) 
9u = W ho h u + b Q (3) 

where Wih is the input-hidden weight matrix, Whh is 
the hidden-hidden weight matrix, Who is the hidden- 
output weight matrix, bh and b Q are bias terms, and 
H is the hidden layer function. In traditional RNNs 
T~L is an elementwise application of the tanh or logis- 
tic sigmoid a(x) = 1/(1 + exp(— x)) functions. How- 

1 For simplicity we assume the transcription sequence 
to be the same length as the input sequence; however this 
may not be true, for example if the transcription network 
uses a pooling architecture (LeCun et al., 1998) to reduce 
the sequence length. 
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ever we have found that the Long Short-Term Mem- 
ory (LSTM) architecture (Hochreiter & Schmidhuber, 
1997; Gers, 2001) is better at finding and exploiting 
long range contextual information. For the version of 
LSTM used in this paper H is implemented by the 
following composite function: 

a n =a (W ia i n + Whahn-! + W sa Sn-i) (4) 
P n = a (W if3 i n + W h ph n _ x + WspSn-x) (5) 

Sn = PnSn-l + OL n tanh (W is l n + Whs) (6) 

7 n = a (W i7 i n + W hl h n - X + W S7 s n ) (7) 
h n = 7n tanh(s n ) (8) 

where a, /?, 7 and 8 are respectively the mp^£ gate, 
forget gate, output gate and state vectors, all of which 
are the same size as the hidden vector ft. The weight 
matrix subscripts have the obvious meaning, for exam- 
ple Who. is the hidden-input gate matrix, is the 
input-output gate matrix etc. The weight matrices 
from the state to gate vectors are diagonal, so element 
m in each gate vector only receives input from element 
m of the state vector. The bias terms (which are added 
to a, /?, s and 7) have been omitted for clarity. 

The prediction network attempts to model each ele- 
ment of y given the previous ones; it is therefore simi- 
lar to a standard next-step-prediction RNN, only with 
the added option of making 'null' predictions. 

2.2. Transcription Network 

The transcription network J 7 is a bidirectional 
RNN (Schuster & Paliwal, 1997) that scans the input 
sequence x forwards and backwards with two separate 
hidden layers, both of which feed forward to a single 
output layer. Bidirectional RNNs are preferred be- 
cause each output vector depends on the whole input 
sequence (rather than on the previous inputs only, as 
is the case with normal RNNs); however we have not 
tested to what extent this impacts performance. 

Given a length T input sequence (xi...xt), a 
bidirectional RNN computes the forward hidden 
sequence ( ft 1, . . . , ft t), the backward hidden se- 
quence ( ft 1, ... , /it), and the transcription sequence 
(fi: • • • > /t) by first iterating the backward layer from 
t = T to 1: 



t t = H(W ith + W tt t w + b t ) 



(9) 



then iterating the forward and output layers from t = 1 
to T: 



o t = W-r t h t 

1 ho 1 



+ w tt 



(10) 
(11) 



For a bidirectional LSTM network (Graves & Schmid- 
huber, 2005), H is implemented by Eqs. (4) to (8). For 
a task with K output labels, the output layer of the 
transcription network is size K + 1, just like the pre- 
diction network, and hence the transcription vectors 
ft are also size K + 1 . 

The transcription network is similar to a Connection- 
ist Temporal Classification RNN, which also uses a 
null output to define a distribution over input-output 
alignments. 

2.3. Output Distribution 

Given the transcription vector f t , where 1 < t < T, 
the prediction vector g u , where < u < U, and label 
k G y, define the output density function 



h(k,t,u)=exp {ft+g k u ) 



(12) 



where superscript k denotes the k th element of the 
vectors. The density can be normalised to yield the 
conditional output distribution: 



Pr(/c G y\t,u) = 



ft(fc, t, u) 



To simplify notation, define 



y(t,u) = Pr(^ n+ i|t,ix) 
0(t,u) = Pr(0\t,u) 



(13) 



(14) 
(15) 



Pi(k\tju) is used to determine the transition probabil- 
ities in the lattice shown in Fig. 1. The set of possible 
paths from the bottom left to the terminal node in the 
top right corresponds to the complete set of alignments 
between x and i.e. to the set y* D B^iy). There- 
fore all possible input-output alignments are assigned 
a probability, the sum of which is the total probability 
Vi{y\x) of the output sequence given the input se- 
quence. Since a similar lattice could be drawn for any 
finite y G y* , Pr(fc|£, u) defines a distribution over 
all possible output sequences, given a single input se- 
quence. 

A naive calculation of Vi{y\x) from the lattice would 
be intractable; however an efficient forward-backward 
algorithm is described below. 

2.4. Forward- Backward Algorithm 

Define the forward variable a(t, u) as the probability 
of outputting y[i :u ] during f\\-t] • The forward variables 
for all 1 < t < T and < u < U can be calculated 
recursively using 



a(t, u) = a(t — 1, u)0(t — 1, u) 
+ a(t,u - l)y(t,u - 1) 



(16) 
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3 ^g^p^h 




Figure 1. Output probability lattice defined by 

Pi(k\t, u). The node at t,u represents the probability of 
having output the first u elements of the output sequence 
by point t in the transcription sequence. The horizontal ar- 
row leaving node t,u represents the probability 0(t,u) of 
outputting nothing at (t, u); the vertical arrow represents 
the probability y(t,u) of outputting the element u + 1 of 
y. The black nodes at the bottom represent the null state 
before any outputs have been emitted. The paths start- 
ing at the bottom left and reaching the terminal node in 
the top right (one of which is shown in red) correspond 
to the possible alignments between the input and output 
sequences. Each alignment starts with probability 1, and 
its final probability is the product of the transition proba- 
bilities of the arrows they pass through (shown for the red 
path). 

with initial condition a (1,0) = 1. The total output 
sequence probability is equal to the forward variable 
at the terminal node: 



Pv(y\x) = a(T, U)0(T,U) 



(17) 



Define the backward variable (3(t, u) as the probability 
of outputting y [u+1:U ] during f [t:T] . Then 

/?(*, u) = f3(t + 1, u)0(t, u) + /?(*, u + l)y(t, u) (18) 

with initial condition f3(T,U) = 0(T,U). From the 
definition of the forward and backward variables it 
follows that their product a(t,u)(3(t,u) at any point 
(£, u) in the output lattice is equal to the probabil- 
ity of emitting the complete output sequence if y u is 
emitted during transcription step t. Fig. 2 shows a plot 
of the forward variables, the backward variables and 
their product for a speech recognition task. 

2.5. Training 

Given an input sequence x and a target sequence 
the natural way to train the model is to minimise the 



log-loss C = — lnPr(y*|a?) of the target sequence. We 
do this by calculating the gradient of C with respect 
to the network weights parameters and performing 
gradient descent. Analysing the diffusion of proba- 
bility through the output lattice shows that Pr(y*\x) 
is equal to the sum of a(t,u)f3(t,u) over any top-left 
to bottom-right diagonal through the nodes. That is, 
V n : 1 < n < /7 + T 

Pr(y»= ^u)P{t,u) (19) 

(t,u):t-\-u—n 

From Eqs. (16), (18) and (19) and the definition of C 
it follows that 



dC 



a(t, u) 

dVi(k\t,u) ~ ~Y>i(y*\x) 



/3(t,u + 1) if k = y u+ i 
P(t+l,u) if k = 
otherwise 

(20) 



And therefore 



d A = y V dL dPv(k'\t,u) 
df? t'o k ^y dPl{k%U) 



dC 



EE 



dC dVi{k%u) 
dVr{k%u) dg* 



(22) 



where, from Eq. (13) 
dFi(k'\t,u) dPr(k f \t,u) 



df t k 



--Pi(k%u) [(W-Pr(*M)] 



The gradient with respect to the network weights 
can then be calculated by applying Backpropagation 
Through Time (Williams & Zipser, 1995) to each net- 
work independently. 

A separate softmax could be calculated for every 
Pr (k\t,u) required by the forward-backward algo- 
rithm. However this is computationally expensive due 
to the high cost of the exponential function. Recalling 
that exp(a + b) = exp(a) exp(6), we can instead pre- 
compute all the exp (/(£,«)) and exp(g(?/[ 1:n ])) terms 
and use their products to determine Pr(fc|t, u). This 
reduces the number of exponential evaluations from 
0(TU) to 0(T + U) for each length T transcription 
sequence and length U target sequence used for train- 
ing. 

2.6. Testing 

When the transducer is evaluated on test data, we 
seek the mode of the output sequence distribution in- 
duced by the input sequence. Unfortunately, finding 
the mode is much harder than determining the prob- 
ability of a single sequence. The complication is that 
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Forward Variables Backward Variables Probabilities 



Figure 2. Forward- backward variables during a speech recognition task. The image at the bottom is the input 
sequence: a spectrogram of an utterance. The three heat maps above that show the logarithms of the forward variables 
(top) backward variables (middle) and their product (bottom) across the output lattice. The text to the left is the target 
sequence. 



the prediction function g(y[i :u \) (and hence the out- 
put distribution Pr(fc|t, u)) may depend on all previous 
outputs emitted by the model. The method employed 
in this paper is a fixed-width beam search through 
the tree of output sequences. The advantage of beam 
search is that it scales to arbitrarily long sequences, 
and allows computational cost to be traded off against 
search accuracy. 

Let Pr(y) be the approximate probability of emitting 
some output sequence y found by the search so far. 
Let Pi(k\y,t) be the probability of extending y by 
k G y during transcription step t. Let pref(y) be 
the set of proper prefixes of y (including the null se- 
quence 0), and for some y G pref(y), let Pr (y\y,t) = 
ni=|^|+i Pr (^n|?/[0:n-i]^)- Pseudocode for a width 
W beam search for the output sequence with high- 
est length-normalised probability given some length T 
transcription sequence is given in Algorithm 1. 

The algorithm can be trivially extended to an TV best 
search (N < W) by returning a sorted list of the N 
best elements in B instead of the single best element. 
The length normalisation in the final line appears 
to be important for good performance, as otherwise 
shorter output sequences are excessively favoured over 
longer ones; similar techniques are employed for hid- 
den Markov models in speech and handwriting recog- 
nition (Bertolami et al., 2006). 

Observing from Eq. (2) that the prediction network 
outputs are independent of previous hidden vectors 
given the current one, we can iteratively compute the 
prediction vectors for each output sequence y + k con- 
sidered during the beam search by storing the hidden 
vectors for all i/, and running Eq. (2) for one step 
with k as input. The prediction vectors can then be 



Algorithm 1 Output Sequence Beam Search 
Initalise: B = {0}; Pr(0) = 1 
for t = 1 to T do 

A = B 
B = {} 

for y in A do 

Pr(y)Pr(y|y,t) 

end for 

while B contains less than W elements more 

probable than the most probable in A do 
= most probable in A 
Remove y* from A 
Pr(y*)=Pr(y*)Pr(0|y,t) 
Add y* to B 
for k G y do 

Pr(i/* + fc) = Pr(y*)Pr(%*,t) 
Add y* + k to A 
end for 
end while 

Remove all but the W most probable from B 
end for 

Return: y with highest \ogPr (y)/\y\ in B 



combined with the transcription vectors to compute 
the probabilities. This procedure greatly accelerates 
the beam search, at the cost of increased memory use. 
Note that for LSTM networks both the hidden vectors 
h and the state vectors s should be stored. 

3. Experimental Results 

To evaluate the potential of the RNN transducer we 
applied it to the task of phoneme recognition on the 
TIMIT speech corpus (DAR, 1990). We also compared 
its performance to that of a standalone next-step pre- 
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diction RNN and a standalone Connectionist Tempo- 
ral Classification (CTC) RNN, to gain insight into the 
interaction between the two sources of information. 

3.1. Task and Data 

The core training and test sets of TIMIT (which we 
used for our experiments) contain respectively 3696 
and 192 phonetically transcribed utterances. We de- 
fined a validation set by randomly selecting 184 se- 
quences from the training set; this put us at a slight 
disadvantage compared to many TIMIT evaluations, 
where the validation set is drawn from the non-core 
test set, and all 3696 sequences are used for training. 
The reduced set of 39 phoneme targets (Lee & Hon, 
1989) was used during both training and testing. 

Standard speech preprocessing was applied to trans- 
form the audio files into feature sequences. 26 channel 
mel-frequency filter bank and a pre-emphasis coeffi- 
cient of 0.97 were used to compute 12 mel-frequency 
cepstral coefficients plus an energy coefficient on 25ms 
Hamming windows at 10ms intervals. Delta coeffi- 
cients were added to create input sequences of length 
26 vectors, and all coefficient were normalised to have 
mean zero and standard deviation one over the train- 
ing set. 

The standard performance measure for TIMIT is the 
phoneme error rate on the test set: that is, the 
summed edit distance between the output sequences 
and the target sequences, divided by the total length 
of the target sequences. Phoneme error rate, which 
is customarily presented as a percentage, is recorded 
for both the transcription network and the transducer. 
The error recorded for the prediction network is the 
misclassification rate of the next phoneme given the 
previous ones. 

We also record the log-loss on the test set. To put this 
quantity in more accessible terms we convert it into 
the average number of bits per phoneme target. 

3.2. Network Parameters 

The prediction network consisted of a size 128 LSTM 
hidden layer, 39 input units and 40 output units. The 
transcription network consisted of two size 128 LSTM 
hidden layers, 26 inputs and 40 outputs. This gave 
a total of 261,328 weights in the RNN transducer. 
The standalone prediction and CTC networks (which 
were structurally identical to their counterparts in the 
transducer, except that the prediction network had one 
fewer output unit) had 91,431 and 169,768 weights 
respectively. All networks were trained with online 
steepest descent (weight updates after every sequence) 



Table 1. Phoneme Recognition Results on the 
TIMIT Speech Corpus. 'Log-loss' is in units of bits per 
target phoneme. 'Epochs' is the number of passes through 
the training set before convergence. 



Network 


Epochs 


LOG-LOSS 


Error Rate 


Prediction 


58 


4.0 


72.9% 


CTC 


96 


1.3 


25.5% 


Transducer 


76 


1.0 


23.2% 



using a learning rate of 10 -4 and a momentum of 
0.9. Gaussian weight noise (Jim et al., 1996) with a 
standard deviation of 0.075 was injected during train- 
ing to reduce overfitting. The prediction and trans- 
duction networks were stopped at the point of lowest 
log-loss on the validation set; the CTC network was 
stopped at the point of lowest phoneme error rate on 
the validation set. All network were initialised with 
uniformly distributed random weights in the range [- 
0.1,0.1]. For the CTC network, prefix search decod- 
ing (Graves et al., 2006) was used to transcribe the 
test set, with a probability threshold of 0.995. For the 
transduction network, the beam search algorithm de- 
scribed in Algorithm 1 was used with a beam width of 
4000. 

3.3. Results 

The results are presented in Table 1. The phoneme er- 
ror rate of the transducer is among the lowest recorded 
on TIMIT (the current benchmark is 20.5% (Dahl 
et al., 2010)). As far as we are aware, it is the best 
result with a recurrent neural network. 

Nonetheless the advantage of the transducer over the 
CTC network on its own is relatively slight. This may 
be because the TIMIT transcriptions are too small a 
training set for the prediction network: around 150K 
labels, as opposed to the millions of words typically 
used to train language models. This is supported by 
the poor performance of the standalone prediction net- 
work: it misclassifies almost three quarters of the tar- 
gets, and its per-phoneme loss is not much better than 
the entropy of the phoneme distribution (4.6 bits). We 
would therefore hope for a greater improvement on a 
larger dataset. Alternatively the prediction network 
could be pretrained on a large 'target-only' dataset, 
then jointly retrained on the smaller dataset as part 
of the transducer. The analogous procedure in HMM 
speech recognisers is to combine language models ex- 
tracted from large text corpora with acoustic models 
trained on smaller speech corpora. 
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3.4. Analysis 

One advantage of a differentiate system is that the 
sensitivity of each component to every other compo- 
nent can be easily calculated. This allows us to analyse 
the dependency of the output probability lattice on its 
two sources of information: the input sequence and 
the previous outputs. Fig. 3 visualises these relation- 
ships for an RNN transducer applied to 'end-to-end' 
speech recognition, where raw spectrogram images are 
directly transcribed with character sequences with no 
intermediate conversion into phonemes. 

4. Conclusions and Future Work 

We have introduced a generic sequence transducer 
composed of two recurrent neural networks and 
demonstrated its ability to integrate acoustic and lin- 
guistic information during a speech recognition task. 

We are currently training the transducer on large-scale 
speech and handwriting recognition databases. Some 
of the illustrations in this paper are drawn from an 
ongoing experiment in end-to-end speech recognition. 

In the future we would like to look at a wider range 
of sequence transduction problems, particularly those 
that are difficult to tackle with conventional algo- 
rithms such as HMMs. One example would be text-to- 
speech, where a small number of discrete input labels 
are transformed into long, continuous output trajecto- 
ries. Another is machine translation, which is partic- 
ularly challenging due to the complex alignment be- 
tween the input and output sequences. 
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Figure 3. Visualisation of the transducer applied to end-to-end speech recognition. As in Fig. 2, the heat 
map in the top right shows the log-probability of the target sequence passing through each point in the output lattice. 
The image immediately below that shows the input sequence (a speech spectrogram), and the image immediately to the 
left shows the inputs to the prediction network (a series of one- hot binary vectors encoding the target characters). Note 
the learned 'time warping' between the two sequences. Also note the blue 'tendrils', corresponding to low probability 
alignments, and the short vertical segments, corresponding to common character sequences (such as 'TH' and 'HER') 
emitted during a single input step. 

The bar graphs in the bottom left indicate the labels most strongly predicted by the output distribution (blue), 
the transcription function (red) and the prediction function (green) at the point in the output lattice indicated by 
the crosshair. In this case the transcription network simultaneously predicts the letters 'O', 'U' and 'L', presumably 
because these correspond to the vowel sound in 'SHOULD'; the prediction network strongly predicts 'O'; and the output 
distribution sums the two to give highest probability to 'O'. 

The heat map below the input sequence shows the sensitivity of the probability at the crosshair to the pixels in 
the input sequence; the heat map to the left of the prediction inputs shows the sensitivity of the same point to the 
previous outputs. The maps suggest that both networks are sensitive to long range dependencies, with visible effects 
extending across the length of input and output sequences. Note the dark horizontal bands in the prediction heat map; 
these correspond to a lowered sensitivity to spaces between words. Similarly the transcription network is more sensitive 
to parts of the spectrogram with higher energy. The sensitivity of the transcription network extends in both directions 
because it is bidirectional, unlike the prediction network. 



